What

SamplingParams._setup() crashed trtllm-serve with std::bad_cast whenever
the tokenizer's eos_token_id is a list instead of a scalar int.

Before:

# sampling_params.py – _setup()
self.end_id = tokenizer.eos_token_id   # could be a list → C++ crash

After:

eos_token_id = tokenizer.eos_token_id
if isinstance(eos_token_id, list):
    self.end_id = eos_token_id[0] if eos_token_id else None
    # register remaining tokens so generation still halts on all of them
    for tok in eos_token_id[1:]:
        if tok not in (self.stop_token_ids or []):
            self.stop_token_ids = (self.stop_token_ids or []) + [tok]
else:
    self.end_id = eos_token_id

Why

HuggingFace tokenizers for models like Llama 3.1 expose
eos_token_id = [128001, 128009] (a List[int]).
_setup() blindly assigned this list to self.end_id, which is typed
Optional[int]. Downstream, base_worker.py passes end_id directly into
tllm.Request(end_id=...) — a C++ nanobind binding that expects int32_t.
PyBind/nanobind cannot cast a Python list to int32_t, so it raises
std::bad_cast, killing the server for every inference request on
any model whose tokenizer advertises multiple EOS tokens.

Why is this PR open if issue #11625 is closed?

Issue #11625 was closed by a maintainer who was unable to reproduce the
crash on their own hardware ("I'm closing this issue. Please feel free to open a new one if you need~"). It was not closed because a fix was merged.
The original reporter confirmed the crash persisted after the closure.
upstream/main still contains the unfixed assignment as of this PR.

The root cause is version-dependent: newer versions of transformers
(≥ 4.43) began returning List[int] from tokenizer.eos_token_id for
models whose tokenizer_config.json has a list-valued eos_token_id.
Earlier versions always returned a scalar, which is why the bug was not
universally reproducible.

How

• Read tokenizer.eos_token_id into a local variable first.
• If it is a list, use eos_token_id[0] as the scalar end_id and
  append eos_token_id[1:] to stop_token_ids so that generation
  terminates on all configured EOS tokens, not just the first one.
• If it is already a scalar, behaviour is identical to before.
• The existing generation_config.eos_token_id list-normalisation logic
  (lines 436-447) is untouched; de-duplication between the two paths is
  handled by the existing stop_token not in self.stop_token_ids guard.

Change is minimal and isolated to the 14 new lines inside _setup().
No other call sites, no architecture changes.

Test

Before fix (reproduction):

from tensorrt_llm.sampling_params import SamplingParams

class FakeTokenizer:
    eos_token_id = [128001, 128009]  # Llama 3.1 style
    pad_token_id = 0

params = SamplingParams()
params._setup(FakeTokenizer(), None, None)
# self.end_id is now [128001, 128009] → std::bad_cast when passed to C++

After fix: params.end_id == 128001 (int), params.stop_token_ids == [128009].

Regression tests added in tests/unittest/llmapi/test_sampling_params.py
(pure Python, no GPU or model weights required):

All pre-commit hooks passed (CRLF, ruff, ruff-format, codespell, DCO).

Summary by CodeRabbit

• Bug Fixes

  • Improved handling of end-of-sequence tokens to support tokenizers with multiple EOS token IDs.

• Tests

  • Added unit tests for SamplingParams initialization and token configuration behavior.